%md-sandbox

<div style="text-align: center; line-height: 0; padding-top: 9px;">

  <img src="https://databricks.com/wp-content/uploads/2018/03/db-academy-rgb-1200px.png" alt="Databricks Learning" style="width: 600px; height: 163px">

</div>

%md

##![Spark Logo Tiny](https://files.training.databricks.com/images/105/logo_spark_tiny.png) Revenue Insights of a Sports Retailer

%md

This is a **Capstone** project, which is design to validate your learnings from the **Just Enough Scala for Spark** Course. 

This Capstone presents you with an opportunity, where you can apply all your learnings including implmentation of classes, methods, exception handling, collections and functional programming to a large problem.

%md

### ![Spark Logo Tiny](https://files.training.databricks.com/images/105/logo_spark_tiny.png) Problem Description

A sports retailer computes revenue performance of a company every end of the year.

The revenue related information is available in comma separated value format. All quarterly revenue information is available as a string, where each substring separated by a **;** contains information about a specific product category in a quarter (in a financial year). 

Each substring contains following information as  **comma** separated values. 

- which quarter of the year (Q1/Q2/Q3/Q4 etc.)

- name of the product category

- total revenue generated in the financial quarter.

**A sample data string is shown below.**

```

 "Q1-2018   Exercise_Fitness    10.33, 

 Q1-2018   Outdoor_Play_Equipment  7.85, 

 Q1-2018   Winter_Sports   3.45"

```

Where each substring separated by **;** is considered a **record**. In the subsequent section, all references to **record** means this substring (e.g. *Q1-2018,Exercise_Fitness,10.33*).

%md

This capstone asks the participants to develop the module in 8 steps.

1. **Step 1:** Initialize the string 

2. **Step 2:** Parse and Validate a record (substring) to figure out if the substring is properly formatted or not.

3. **Step 3:** Define a method to filter good and bad records from the complete string.

4. **Step 4:** Define a case class to represent a good record and convert all good records to an array of case classes

5. **Step 5:** Create a Class that encapsulates all good records and provides methods to calculate insights

    - the total revenue generated by the sports company

    - which takes a product category and returns the total revenue generated

%md

### ![Spark Logo Tiny](https://files.training.databricks.com/images/105/logo_spark_tiny.png) Step 1: Initialize the String

The string that contains the data is given below. Store the string in a variable named **RevenueInfo**.

```

 "Q1-2018,Exercise_Fitness,10.33; 

  Q1-2018,Outdoor_Play_Equipment,7.85; 

  Q1-2018,Winter_Sports,3.45;

  Q2-2018,Exercise_Fitness,7.63; 

  Q2-2018,Outdoor_Play_Equipment,5.05; 

  Q2-2018,Winter_Sports,-;

  Q3-2018,Exercise_Fitness,1.31; 

  Q3-2018,Outdoor_Play_Equipment,3.95; 

  Q3-2018,Winter_Sports,1.50;

  Q4-2018,Exercise_Fitness,5.71; 

  Q4-2018,Outdoor_Play_Equipment,6.52; 

  Q4-2018,Winter_Sports,4.15"

```                     

%md

### ![Spark Logo Tiny](https://files.training.databricks.com/images/105/logo_spark_tiny.png) Step 2: Parse and Validate a record

In this step, validate a substring that represents a record e.g. **"Q1-2018,Exercise_Fitness,10.33"**

1. Define a method **validateRecord**, which takes a string named **recStr** (representing a record)

2. Parse the string by **comma**

3. Validate that the string contains 3 fields 

   - If there are less than 3 fields then throw an exception

4. Validate if the thrid field is numeric or not (Note: try type casting to float) 

   - If not numeric then throw an exception

5. Surround the complete implementation by **`try`**/**`except`** to handle exceptions

6. If the record is valid (i.e. passes both the checks), 

   - then return a tuple with tag (GOOD) and the record string as ("GOOD", recStr)

   - if the record is not valid then return a tuple with tag (BAD) and the record string as ("BAD", recStr)

   Note: **recStr** is the original string passed to the method

%md

### ![Spark Logo Tiny](https://files.training.databricks.com/images/105/logo_spark_tiny.png) Step 3(a): Define a method to filter good and bad records

In this step parse the string **RevenueInfo** into substrings and use the method **validateRecord** to validate each substring.

1. Define a method called **parsetData**, which takes the complete string stored in **RevenueInfo**

2. Parse the string by a **semicolon(;)**  character, which returns an **Array** of substrings

3. Iterate through all the substrings (using functional programming features), validate each substring

4. Filter all good records that contains the tag **GOOD**

5. Filter all bad records that contains the tag **BAD**

6. Return a tuple that contains good and bad records as  **(array of good records, array of bad records)**

%md

### ![Spark Logo Tiny](https://files.training.databricks.com/images/105/logo_spark_tiny.png) Step 3(b): Filter good and bad records

1. Invoke the above method **parseData** with **RevenueInfo** as a parameter.

2. Store the returned tuple values (containing arrays of good and bad records) into two variables called **goodrecs** and **badrecs**.

%md

### ![Spark Logo Tiny](https://files.training.databricks.com/images/105/logo_spark_tiny.png) Step 4 (a): Define Named Tuple to represent each good record

1. Define a named tuple **CategoryQuarterlyRecord** with three fields 

  - **quarter**  (String)

  - **category** (String)

  - **revenue** (Float)

%md

### ![Spark Logo Tiny](https://files.training.databricks.com/images/105/logo_spark_tiny.png) Step 4(b): Define an utility method to convert a substring into named tuple

1. Define a method **convertToClass**, which

  - Takes the **string** representing a good record ( e.g. *"Q1-2018,Exercise_Fitness,10.33"*) 

  - Parses the string by **comma** 

  - Returns a case class of type **CategoryQuarterlyRecord**